Philosophy of IR Evaluation

• System evaluation: how good are document rankings? • User-based evaluation: how satisfied is user? NIST Why do system evaluation? • Allows sufficient control of variables to increase power of comparative experiments – laboratory tests less expensive – laboratory tests more diagnostic – laboratory tests necessarily an abstraction • It works! – numerous examples of techniques developed in the laboratory that improve performance in operational settings NIST Cranfield Tradition • Laboratory testing of retrieval systems first done in Cranfield II experiment (1963) – fixed document and query sets – evaluation based on relevance judgments – relevance abstracted to topical similarity • Test collections – set of documents – set of questions – relevance judgments NIST Cranfield Tradition Assumptions • Relevance can be approximated by topical similarity – relevance of one doc is independent of others – all relevant documents equally desirable – user information need doesn't change • Single set of judgments is representative of user population • Complete judgments (i.e., recall is knowable) • [Binary judgments] NIST The Case Against the Cranfield Tradition • Relevance judgments – vary too much to be the basis of evaluation – topical similarity is not utility – static set of judgments cannot reflect user's changing information need • Recall is unknowable • Results on test collections are not representative of operational retrieval systems